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ABSTRACT. Let O = Hi \ 02, where Oj and Q2 are two smooth bounded pseudoconvex 
domains in C n ,n > 3, such that D.2 C Cl\. Assume that the 3-Neumann operator of Qi 
is compact and the interior of the Levi-flat points in the boundary of 02 is not empty (in 
the relative topology). Then we show that the Hankel operator on Q with symbol <p, H^, is 
compact for every <p G C(f2) but the 3-Neumann operator on Cl is not compact. 

Let O be a domain in C" and A 2 (Q) denote the Bergman space on O, the space of square 
integrable holomorphic functions on Q. The Bergman projection, the orthogonal projec- 
tion from L 2 (Q) onto A 2 (Q), is denoted by P Q and the Hankel operator with symbol 
<p e L°°(fi), denoted by H$, is defined as H$(f) = <pf - P n {<pf) for / e A 2 (Q). 

The 9-Neumann problem is solving Dm = v where □ = 99 + 9 9, on square integrable 
(0, l)-forms and 9* is the Hilbert space adjoint of 9. We will denote the solution operator to 
□ on a domain Q, the 9-Neumann operator on O, by N n . On bounded pseudoconvex do- 
mains, Hormander HH6r65B showed that N is a bounded operator on L 2 Q ^ (O), and Kohn 



HKoh631 showed that P n = I - 9*N n 9. Therefore, Hjj*(f) = d*N n (fd(p) for / e A 2 (Q) and 
(p eC 1 (O) . We refer the reader to IIClMllSlrTOll for more information about the 9-Neumann 



problem and to |C§09I (and references therein) for more information on compactness of 
Hankel operators on Bergman spaces. 

Given Kohn's formula, it is natural to expect strong connections between N and Hankel 
operators on A 2 (Q). For example, if O is bounded and pseudoconvex, and N° is compact 
on L? ^ (O), then is compact on A 2 (Q) for all <p G C(O) (see flHasOSi Theorem 3] and 



nStrlOl Proposition 4.1]). We are interested in the converse, which is a question of Fu and 



Straube [FSOl, Remark 2]: does compactness of on A 2 (Q) for all <p G C(O) imply that 
N n is compact on L 2 Q ^ (O)? The answer to this question is still open in general. However, 
if O is allowed to be non-pseudoconvex we can show that the answer is no (see Theorem [l] 
below). 
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We call O an annulus type domain if Q = Ox \ O2 where Oi and 02 are smooth, 
bounded, pseudoconvex, and O2 C Oi. The following theorem of Shaw, contained in 



llShalOl Theorem 3.5], guarantees that the 9-Neumann operator exists on annulus type do- 
mains in C n for n > 3 and it is connected to the Bergman projection the same way as it is 
on bounded pseudoconvex domains. 

Theorem (Shaw). Let O = Oi \ D.2, where Cli and CI2 are two smooth bounded pseudoconvex 
domains in C n , n > 3, such that CI2 C Oi. Then 

i. N^ 1 exists on TjL Jd)for 1 < q < n — 2, 

ii. d*N Cl is the canonical solution operator for d, 

iii. P n = I - a*N n a. 



In fact Shaw ( [ShalOl Theorem 3.5]) showed that the 9-Neumann operator is bounded 
on (0, l)-forms for n > 2. However, the space of harmonic forms ^, defined in the next 
section, is infinite dimensional when n = 2 and trivial when n > 3. Hence, when n = 2 
items ii. and iii. in Shaw's theorem above are not valid. 

The following theorem is our main result. We note that B(p,r) denotes the open ball 
centered at p with radius r, and a point p, in the boundary of a smooth domain Q C C n , 
is called Levi-flat if the Levi form of O, the restriction of the complex Hessian of a defining 
function onto complex tangent space, is constant zero at p. We denote the boundary of a 
domain O by fcO. 

Theorem 1. Let O = Oi \ Q2, where Oi and CI2 are two smooth bounded pseudoconvex domains 
in C n ,n > 3, such that CI2 C fii. Assume that the d-Neumann operator N ni is compact on 
L? 01 x(Oi) and that there exists a ball, B(p,r) centered p G &Q2 with radius r > such that 
B(p, r) n bd:2 is a Levi-flat surface. Then the Hankel operator is compact on A 2 (Q) for every 
(p G C(O) but the d-Neumann operator N° is not compact on L^ Q ^ (O). 

See Remark |3]f or an explanation of why we stated the above theorem for domains in C w 
for n > 3. 

Remark 1. Hankel operators are closely connected to a very important class of operators 
called Toeplitz operators. The Toeplitz operator on A 2 (0) with symbol <p G L°°(n), de- 
noted by T$, is defined as Tjj*f = P n ((pf) = (pf ' - Hftf for / G A 2 (Q). Let (p G C(Q) such 
that (p(z) 7^ for z G bCi\. Choose i/?, <p\ G C(O) such that <p\{z) = <p{z) for z G bCl\ and 
I > on O, and xp = \/<p\. Then 

\ l $ - L $ x + - <p — (pi + L <p\ip ~ 1 M <pi H ip — 1 + K 
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where K = T^^T^ — P^M^H^. Now assume that 0,01, and O2 are as in Theorem [U 
Then K is a compact operator (TP ^ is compact because <p\ — <p = on the outer bound- 
ary of O and is compact by Theorem [T]). Hence, T^ is Fredholm for any (p G C(O) 
with the property that cp(z) 7^ for z G bOi. Fredholm property of Toeplitz operators on 
some pseudoconvex domains in C" has been studied by several authors (see, for example, 
IIVen72llHT97l V 

Remark 2. Hankel operators can also be expressed as commutators of the Bergman projec- 
tion with multiplication operators. These commutators proved to be useful in the proof of 
the complex version of Hilbert's seventeenth problem (see [CD97J). For more information 
about relations between the commutators and the 9-Neumann problem we refer the reader 
to UStrlOi Chapter 4.1]. The computation is as follows: 

(P n (<Pg),h) L 2 {n) = (<Pg,h) L 2 {n) = (g,H^h) L2{n) = ((H%)* 8 ,h) LHn) 

for (p <E L°°(Cl),h e A 2 (0), and g±A 2 (0). Hence for any / G L 2 (0) we have 

[M<p, P n ] (/) = [M<p, P n ] (P n /) + [Afy, P n ] ((/ - P°) /) 
= H$P n f - P n M<p(I - P n )/ 
= H?P n /-(H^)*(I-P n )/. 

When / G A 2 (0),P°/ = /, and (I - P n )/ = 0, whence H$ = [M^P ] on A 2 (0). Note 
that (H^)* : L 2 (0) -4- A 2 (0) and it is compact if and only if is compact. Therefore, 
H^p is compact on A 2 (0) if and only if [M^,, P ] is compact on L 2 (0). We note that similar 
calculations as well as related issues appeared in [Has08] on pseudoconvex domains (see 
also ltCD97l[FS0TH ). 

Corollary 1. Let O, Oi, and O2 be as in Theorem^ Then the commutator [M ( p,P ri ] is compact 
on L 2 (0) for every cp G C(O) but the d-Neumann operator N n is not compact on L 2 Q ^ (O). 

Example 1. Here, we give an explicit example. Let Ai(f) = for t < and Ai(f) = e -1 ^ for 
t > and 

A(zi,z 2 ,z 3 ) = A 1 ( |zi| 2 - \ j + Ai ( |z 2 | 2 - \ J +Ai ( |z 3 | 2 - \ j - e" 3 . 



4/ 1 V 4/ 1 V 4 , 

One can check that Ai is a convex function on (—00, 1/2). Let us define 

O = |(zi,z 2 ,z 3 ) G C 3 : |zi| 2 + |z 2 | 2 + |z 3 | 2 < 9 and A(zi,z 2 ,z 3 ) > oj . 

So Oi = B(0,3),O 2 = {(zi,z 2 ,z 3 ) G C 3 : A(zi,z 2 ,z 3 ) < 0} , and O = Oi \ 2 is a smooth 
bounded annulus type domain in C 3 . By construction bCl n B((0,0, \/7/\/l2), 1/3) is a 
Levi-flat surface. Then Theorem[T]and Corollary [1] imply that [M^, P ] is compact on L 2 (0) 
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for every (p G C(O) (hence H^ 1 is compact on A 2 (Q) for every ^ £ C(Q)) but N° is not 
compact on L 2 Q ^ (Q). 

Proof of Theorem Q] 

Let 

Hf 01) = ker(D n ) = {u e Dom(8) n Dom(9*) : du = 0, Tu = 0} C L? 01) (Q). 

We call 1} the space of harmonic (0, 1) -forms and denote the orthogonal projection 
from L 2 Q ^ (O) onto %Q ^. The following Lemmas will be useful in the proof of Theorem[l] 

Lemma 1. Lef O be an annulus type domain in C n for n > 2. Then N n is compact on L 2 ^(O) 
z/anrf onZy zf/or euery £ > f/We exz'sfs C £ > swc/z f/iaf 

(1) ||u|| 2 < e (||du|| 2 + ||a*u|| 2 ) + ||H n M|| 2 + C e ||i< - H n uf_ ! 

/or u G Dora (3) n Dora (9* ) C L 2 01) (O). 

Proo/. We note that 9 has closed range in L 2 Q ^ (Q) (see UShalOl Theorem 3.3]). Let us define 

r = Dom(a) n Dora(a*) n (ji^ rl) ) ± 

(X- 1 - denotes the orthogonal complement of X) and equip the space T with the graph norm. 
That is, ||w|| 2 = \\du\\ 2 + ||9*u|| 2 . Then the embedding; : Y ^ L 2 01 ^(O) is continuous 
llShalOl . Furthermore, N — j of (for a proof of this see IIStrlQl Theorem 2.9]. Although 
pseudoconvexity is assumed in MStrlOi Theorem 2.9] its proof applies in our situation as 
well because / is a bounded operator). Hence N is compact if and only if / is compact and 
compactness of j is equivalent to the following estimate ( UStrlOl Proposition 4.2]): for all 
e > there exists C £ > such that 

\\u\\ 2 < £ (||9w|| 2 + ||9*w|| 2 j + C £ ||u|| 2 1 for tier. 

One can substitute u — H^u instead of u above to show that the inequality above is equiv- 
alent to ©. □ 

Lemma 2. Let O be an annulus type domain in C n for n > 3 such that N n exists and it is 
compact on L 2 Q ^ (Q). Let p be a boundary point of O and r > such that U = O n B(p, r) is a 
pseudoconvex domain. Then N u is compact on L 2 Q ^ (IT). 

Proof We note that since n > 3 the space %Q ^ is trivial and the proof is essentially con- 
tained in flStrlOl Proposition 4.4] once we know that 7^ ^ is trivial. However, we will give 
the proof here for the convenience of the reader. 
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Lemma [T] implies that compactness of N n is equivalent to the following estimate: for all 
e > there exists C £ > such that 

— 1 1 ii — ^ 1 1 \ ii 1 1 o , — v , — . 

IMIn < e(||o«|ln + ||o u \\ri) + n ror u e Dom(d) n Dom(a ). 

II l\2 2 

\\Z — P — i 

Let A e (z) = — . One can check that — 2r + e < A £ (z) < 2r + e and 

£ 

A 9 2 A £ (z)^ 1 



for z G V £ = {z € C M | dist(z, bB(p,r)) < e} for £ G C M . Now we choose (p £ as a smooth cut- 
off function (0 < (p e < 1), (p E = 1 near bB(p,r) and supported in V E . The triangle inequality 
implies that 

(2) ||M||ff < 2||4> £ «|| 2 J + 211(1-^)^11^ for u G L( 0/1) (U). 

Letu G Dom(a) nDom(9*) C L 2 01) (l?) then (1 - <p £ )u G Dom(9) n Dom(9*) C L 2 01) (O). 
Since the domain IT is not C 2 -smooth a direct application of Morrey-Kohn-Hormander for- 
mula is not possible. However, one can use the Morrey-Kohn-Hormander formula (with 
weight A £ + xp) with the exhaustion procedure developed in [Str97j (see also flStrlOl Corol- 
lary 2.15]) together with the fact that (p £ u belongs to the domain of 9 on U to show that 

(3) UeU\\ 2 U < e(\\d(cpeU) \\ 2 U + \\T (<p e u) f u ). 

In the inequality above we used generalized constants. That is, A < B denotes that A < cB 
where c > is independent of quantities of interest. Thus, from (O and © we get 

\W\\u Z e (P (<M) IIlt + 113* (<M) fu) + 11(1 - 

<e(||aw|| 2 J + ||a*M||u+ IKv^mI^) + ||(i-fc)u|&. 

(1 — <p e )u and V^ £ w can be viewed as forms on O in Dom(9) n Dom(9*). Let us choose 
Xe £ C~(B(p, r)) such that ^ £ = 1 on the union of the support of V(p e and support of 
1 — (p £ . Then 

(4) II"IIlt ^ e (ll^||u+ + c ^lk £ M llLr 

for u G Dom(9) n Dom(3*) C L 2 Q ^ (U). Now, we will try to estimate the last term in (|4j). 
We note that XeU G Dom(9) n Dom(cT) C L 2 ^(O) for any u G Dom(9) n Dom(9*) C 
L 2 Q ^ (U). Compactness of N n on L 2 Q ^ (O) implies that for all e' > there exists C £ / > 
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such that 

\\Xeu\\h ^ e ' (ll%£ w )lln + l|d*Cfc")lln) + c £ 'll^ w ll-i,n 

(5) < e'dl^wlln + + II V^Hn) + C £/ \\u\\ 2 _ hU 
Now, let us estimate || V^ £ u||q in © 

II V^ E "||n < II v ^"ll?z ^ c e\W\\u ^ Q (||a«||u + ||3*«||u) ■ 

In the last step we used the basic estimate on U, \\u\\fj < C(||3m||^+ ||d u\\u)- 

Thus, combining © and the discussion after it with dU) we get a compactness estimate 
on U. That is, for all e > there exists C £ > such that 

(6) IMIu % e (\\M\u + \\d* u \\u) + C e\\ u \\-l,U 

for all « <E Dom(a) n Dom(9*) C L? Q 1} (IT). □ 

Remark 3. We chose the domain O in C w for n > 3 because we do not know if the localiza- 
tion in the proof of Lemma |2] is possible when n = 2. If Q C C 2 is an annulus type domain 
then U9 Q l) is an infinite dimensional space UShalOl Theorem 3.5] whereas H^q^ is trivial 



when O is pseudoconvex. 

In the following Lemma Ry denotes the restriction operator onto V. 

Lemma 3. Let Q = \ O2 where Oi and CI2 we two smooth bounded pseudoconvex domains in 
C n ,n > 3, such that CI2 C Oi and <p G C 1 (Oi). T/zen H^ 1 is compact on A 2 (Oi) z/and onZy if 
Hrcl(4>) zs com V ac t on A 2 (Q). 

Proof. Let us prove the necessity first. By Hartogs extension theorem there exists a unique 
bounded extension operator Eq 1 : A 2 (fl) — > A 2 (Hi). One can check that RnH^E^f 
solves du = fdcp on O. Furthermore, since ^R n ^\f is the canonical solution (the solution 
with minimal L 2 norm) for du = fdcp we have 



(I-P n )R n H^E% =H n 



Therefore, compactness of 1 on A 2 (Hi) implies that ^ is compact on A 2 (Q). 
To prove the converse assume that H5? is compact on A 2 (Q) and U be a neighbor- 



hood of p G &Oi such that IT n O4 = LTnOisa domain. Then i. in [C§09, Proposition 1] 
implies that H^ nU , ,R nnU is compact on A 2 (H). We note that even though [ £§09| Propo- 



sition 1] is stated for pseudoconvex domains, i. is still true for general domains. However, 

jnnu 7? _ irfiin!! 
l Rnnu(4>) KnnU ~ M R ni nuW 



one can check that (I - P ninU )H^ j(d)) Rnnu = H^ n " R ni nu on A 2 ^) and hence 
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H R ^ n(j ^- ) ^n 1 n(J is compact on A 2 (Qi). Now ii. [ C§09 Proposition 1] implies that 1 is 



compact on A 2 (Oi). □ 

Remark 4. We note that compactness of N Ql on A 2 Q ^ (Oi) is equivalent to compactness of 
N Q on A 2 ^ (O). This can be seen as follows: 

Range's formula, N n * = (Tn^ 1 ) (^N" 1 )* + (a*N n i)* (d*N n ^ , together with the 
fact that (a*N 2 ni )* u = for u G A^Oi) imply that N n iu = (a*N n i)* (a*N n i) u for 
w G A^ 01 ^(Oi). Hence, N Ql | A 2 is compact if and only if 9*N ni |^2 j-q^ is compact. 
(Here f\x denotes the restriction of the operator / onto the space X). Similarly, one can 
show that N°|^2 ^ is compact if and only if 9*N°| A 2 ^ is compact. On the other 

hand, Lemma |3] implies that compactness of 9*N fil | A 2 (q \ is equivalent to compactness 

of5>% ?0J)( n, 



We will need the following theorem of Catlin. For a proof we refer the reader to the proof 
of Proposition 9 in [FS01J (see also |§S06]). We note that even though Catlin's Theorem in 



IFS01J is stated in C 2 , the same proof works for the following version in C n . 

Theorem (Catlin). Let Cibea bounded pseudoconvex domain in C n , n > 2, with Lipschitz bound- 
ary. Assume that bCl contains an (n — 1) -dimensional complex manifold. Then N° is not compact 
on L 2 04) (O). 

Proof of Theorem^ The assumption that N^ 1 is compact implies that 1 is compact for all 
<p G C a (ni) (see llFSOTl Proposition 4], llStrlOl Proposition 4.1], and IIHas08i Theorem 3]). 
Since any <p G C 1 (Q) can be extended as C 1 function on C w Lemma [3] implies that is 
compact for all <p G C 1 (Q). However, one can approximate any cp G C(O) uniformly on Q 
by C 1 functions. Therefore, we conclude that is compact on A 2 (Q) for all <p G C(O). 

Now we will show that N n is not compact. Shaw's Theorem, stated in the introduc- 
tion, implies that N° is a bounded operator on L 2 Q ^ (O). Assume that N n is compact on 
L 2 01 ^(O). Let us choose p G bD.2 and r > so that IT = Q Pi B(p,r) is a domain that 
does not intersect bCl\ and the (inner) boundary of O in B(p,r) is Levi-flat. Hence U is 
a non-smooth bounded pseudoconvex domain. Lemma |2] implies that if N n is compact 
on L 2 01 ^(O) then N u is compact on L^ Q1 ^(U). Compactness of N u implies that 9 has a 
compact solution operator on L 2 Q ^ (IT). However, this contradicts Catlin's Theorem stated 
above. Hence, N n is not compact on L 2 Q ^ (O). This contradiction with the assumption that 
N is compact completes the proof. □ 
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